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ABSTRACT 



This paper reviews recommended guidelines for World Wide 
Web-based instruction and provides a framework for libraries worldwide in 
designing Web-based instructional programs. The first section describes 
Web-based training, including interactivity and the content and objectives of 
Web-based library tutorials. The second section discusses Web design and 
evaluation criteria for library professionals, including guidelines developed 
by the Library Instruction Round Table of the American Library Association 
and the Instruction Section Teachings Methods Committee of the American 
Association of College and Research Libraries. The third section addresses 
Web design and evaluation criteria in non-library sources. The last section 
covers ssessment and usability studies. An addendum provides brief 
descriptions and Web site addresses of projects outside the United States 
that are involved with teaching and learning in a networked environment. 
(Contains 14 references.) (MES) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



05 77^S ED 441 454 



http://www.ifla.org/IV/ifla65/papers/102-163e.htm 



Evaluation of web-based library in...nference Programme and Proceedings 

• - # 



^ 




ffff A AfCTT* 


Search Contacts 


IFLANtT- 


Intern ailonal Fed € ratio ii of tlbrafy Associatlonis and Instit ytlons 




Annual Conference 



65th IFLA Council and General 
Conference 

Bangkok, Thailand, 

August 20 - August 28, 1999 

Conference 

Proceedings 

Code Number: 102-163(WS)-E 

Division Number: 11 . r i i r i. \ 

Professional Group: Science and Technology Libraries: Workshop (King Mongkut Institute of Technology Library) 

Joint Meeting with: User Education Library 
Meeting Number: 163 
Simultaneous Interpretation: No 




Evaluation of web-based library instruction programs 



Tess Tobin & Martin Kesselman 

Rutgers University Library Media Services 
Piscataway, New Jersey 



USA 



Paper 

Purpose 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 



A.L. Van Wesermael 



U.S. DEPARTMENT OF EDUCATION 
Office ol Educational Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

id This document has been reproduced as 
received from the person or organization 
originating it. 

Q Minor changes have been made to 
improve reproduction quality. 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

J 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



The World Wide Web (WWW) is changing the way academic libraries teach and learn. Academic 
libraries have embraced the potential of the WWW developing innovative ways to meet the needs 
of users. One of these ways is to deliver automated bibliographic instruction in the form of 
web-based tutorials. This paper will review recommended guidelines for web-based instruction and 
will provide a framework for libraries worldwide in designing web-based instructional programs. 



The stated purpose is to identify important desi^ and evaluation criteria for web-based 
literacy/information instruction. Building on an initial investigation, the author examined library 
instruction literature to see what criteria was available for those wanting to create effective 
web-based training modules. The author also looked at non-library literature to find web-based 
training criteria, which was then compared to the criteria developed by library professionals. Over a 
two-year period, the author was able to note a progression and shift in the guidelines developed by 
library professionals, which reflects, not only, improvements in the technology itself, but a better 
understanding by librarians as to what elements are needed to structure an effective online 
bibliographic instruction program. The library guidelines were examined within the context of 
current research on bibliographic instruction to determine the usability or deficiency of the 
established criteria. 



To date there are very few end-user studies in the library instruction literatine regarding online 
tutorials. The last section of this paper will report on assessment and usability studies of web 
learning modules. 
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Introduction 

Computer scientists have been researching the principals behind effective interface design of 
computer applications since the 1940's. Stover and Zink report that with the emergence of the 
World Wide Web in the early 1990's, organizations of all types are recognizing the importance of 
the World Wide Web as a tool, not only for gaining access to information, but also as a means of 
disseminating information about their activities, products, and services, i 

Academic librarians have endorsed the possibilities of the WWW and first began to use web 
technology to create home pages, as starting points, or gateways for searching for information about 
the library. A home page reflects the institutional character of the university or college and is 
unique in that it gives the library the opportunity to express its own mission and philosophy. 

With the growth of distance learning and the need to offer services to patrons off campus as well as 
in the library, the libraries needed to explore other ways to deliver library information skills. The 
WWW provides a dynamic environment for distributing information over a large network and 
web-based instruction is becoming the desired tool for these new users. In the web environment it 
doesn't matter if the user is connecting to library resources on the computer in the library, elsewhere 
on campus or from his/her home. 

Developing an online module can be a challenge for many libraries. As librarians approach this task 
they should not overlook the groundwork laid by computer scientists in the field of computer 
interface design and computer/human interaction. The knowledge the instruction librarian brings to 
the task gathered from their classroom experiences and the findings from computer-assisted 
learning studies need to be considered when planning the design, content and evaluation of a 
web-based tutorial. 

What is web-based instruction? 

Web-based training (WBT) is an innovative approach to distance learning in which computer-based 
training is transformed by the technologies and methodologies of the World Wide Web (WWW), 
the Internet, and Intranets. 2 It allows self-directed, self-paced instruction in any topic. Vt^T is 
media-rich training fully capable of evaluation, adaptation, and remediation that can provide the 
available tools to organize and deliver content into well-crafted teaching systems. 

WBT has certain distinguishing characteristics, which makes it an ideal tool for bibliographic 
instruction. WBT is an interactive tool. It expands upon computer-based training with activities like 
discussion forums, mail lists and chat sessions. The user sets his/her pace and direction. 

The library instruction literature points to active learning as the best means to deliver bibliographic 
instruction. ^ The interactivity component of WBT is a two-way interaction. The user inputs to the 
system and the system provides feedback that either reinforces the user or provides guidance for 
learning. This can be measured by having the user select the correct response and guide the user to 
continue if an incorrect response is made. WBT can also meet the needs of diverse populations and 
learning styles. The multi-media used with online training modules can help all types of learning 
styles - visual, verbal and aural. 

Web-based training is not a collection of information pages. Presently, an examination of many 
library instruction pages shows that libraries have rushed to embrace this new technology and they 
are using it effectively. Some librarians have created multiple pages full of text with complete 
verbal explanations of the objectives of the desired library skills. These sites are iminteresting and 
do not involve the user. Creating a table of contents with links to full page documents is a misuse of 
the technology and not an interactive learning tool. 

An interactive component such as a quiz or feedback form tests the user's knowledge and allows the 
librarian to determine the effectiveness of the learning. Without this component, the librarian 
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cannot monitor the learning, and thus cannot adapt the module to best suit the learner. Assessing 
the modules and reviewing them to alleviate confusion and frustration will go a long way to help 
keep the learner motivated and more willing to work through the objectives and attain information 
literacy. 

The content and objectives of the web-based tutorials examined for this study varies greatly. Some 
tutorials take the user through the physical layout of the library with maps and pictures. The next 
type of tutorial explains basic library skills and can offer a glossary of library terms. Another type 
of tutorial offers instruction on using the library's OP AC, and leads the user through the steps of 
locating books, magazines and other library materials. To be effective these tutorials should be 
learning modules that promote the institutional needs of the library and relate the information 
library skills the library wises its users to achieve. 

The next level of tutorial presents the more conceptual aspects of information literacy. These are 
tutorials on the research process and evaluation of sources. Some of these tutorials review various 
library networked databases while explaining Boolean and keyword and subject searching 
techniques. Searching the Internet is a focus of many tutorials, along with interactive modules that 
offer guidelines on how to evaluate Internet sources. Each tutorial is different in design, content and 
instruction level, reflecting the personality and nature of each library. 

Web Design and Evaluation Criteria by Library Professionals 

For librarians developing online instructional tutorials to supplement their bibliographic instruction 
programs, the web guidelines developed by the Library Instruction Round Table (LIRT) of the 
American Library Association should be considered. LIRT advocates library instruction as a means 
for developing competent library and information use as a part of lifelong learning. The LIRT 
homepage has links to the Top Ten Instruction Articles for the past three years, and links to Library 
Instruction Tutorials. In July, 1997 the LIRT Research Committee posted a page on Website 
Evaluation Criteria. At that time the three basic elements recommended by LIRT to include in a 
web page were access, design and content. This page has evolved and now is called Web Standards 
and Guidelines. The section on access is no longer an element since web browsers and servers have 
become quicker and more stable. The content and design elements are more brief and list technical 
points. The importance is on elements to avoid rather than offering standards on what makes up a 
good web page. The new addition to the page is "Procedural/Technical." This new element refers to 
HTML coding and resolution requirements and settings. The Web Standards and Guides are useful 
from a technical standpoint and instruction librarians should review them. 

The Instruction Section Teachings Methods Committee,^ a committee of the American Association 
of College and Research Libraries (ACRL) has also prepared web criteria for instruction. This 
committee is charged with providing a forum for librarians interested in both the theoretical and 
practical aspects of teaching methods. These guidelines are intended to help libraries' efforts in 
web-based instruction and are more pedagogical in nature and reflective of the committee's mission 
to have librarians be successful in their instructional endeavors. Since online instruction is a new 
format for many, the guidelines are thorough, and the design standards are in keeping with what 
constitutes a successful web-based training program. These guidelines, updated three times over the 
last two years, are keeping up with advances in the technology. 

The original page included Access, Design and Layout, Content and Audience. Updates have been 
made and the standard on content now states that pages should be tight and concise - they convey 
'just the facts' without excess verbiage. Many libraty tutorials are heavy on text. Web-based 
material should not read like a book. The presentation and delivery of online information have their 
own unique characteristics. Research indicates that the page will either capture the attention of the 
user or it will not. 

The evaluation criteria for design is now divided into two parts. General Design and Layout covers 
some basic design elements to web page development. Large pages should be broken down into 
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discrete pages. Graphics should help to convey information and not just be decorative; each page 
should have a clear title and includes headers and footers. 

The newest standard, Design and Layout of Interactive Tutorial Material, incorporates interactivity 
into the tutorials. Exercises, which provide feedback and test whether the student has grasped the 
concepts, are needed. This supports the learning theory behind bibliographic instruction that the 
modules need to engage and promote active learning. Navigational mechanisms which allow 
learners to repeat or move around within the tutorial have proven to be more effective ways to 
present information so it is not only accessible, but it is retained. 

Last but not least, the criteria considers the audience as a necessary element. The content and links 
should be suitable to the user. The site should address a particular facet of library instruction. This 
would mean that the page should clearly state the objectives and goals of the instruction imit being 
presented. It is important to note that the IS Teaching Methods Committee referred to the LIRT 
Web Standards along with the guidelines and evaluation criteria used by Argus Clearinghouse^, 
Cyberstacks^and the Reference Collection Development: A Manual ^ in selecting and revising their 
criteria. 



Another ACRL committee, the Instruction Section Emerging Technology Committee - has acted 
more as an information sharing vehicle and facilitator for librarians interested in sharing 
instructional materials that they have designed in a networked environment. One of the cormnittee's 
projects is the ACRL/CNI Internet Education Projectl^, which has developed selection criteria used 
to review and evaluate instructional materials libraries wish to post on the web. The ten criteria 
statements require that the information is accurate, the material, content and language is clear, the 
material provides instruction using technology, the material makes effective use of good web 
design (graphics, examples, interactive elements) and the material enhances the learning 
experience. The committee has also created a Training Resources Bookmark^- with links to 
numerous sites on web education, library instruction research and web design. 

Web Design and Evaluation Criteria in Non-Library Sources 

Business organizations delivered instructional models via the web long before the library 
commimity. Large corporations, which have spent millions on training, have found more 
cost-effective and successful ways of teaching employees using this platform. Their experience and 
research in this arena is valuable to librarians planning web-based tutorials. There are many sites 
that cover web design for training, but there are two worth mentioning for this study. 

Walt Howe has posted a series of articles called Web Page Quality!^ including Evaluating Quality, 
Producing Quality Web Page Content, and Creating Web Page Design which have been covered in 
the library guidelines. Test Your Pages is an important component that does not appear in any of 
the library guidelines up to now. 

Another site useful for this study is the WBT Information Center!^. The section Primer has rules for 
Good Web Design and The Development Process guides one through the essential steps for 
developing web-based training from user analysis, interface design, usability testing, instructional 
design, scripting and coding. The Center also has links to Web Style Guides, Discussion Groups 
and Articles and Position Papers on web-based training. This site is attractive and is one stop 
shopping for those interested in creating web instruction guidelines and materials. 



Assessment and Usability Studies 

Evaluating instructional programs is essential. This appears to be the step lacking from the library 
guidelines and is absent in 90% of the library tutorials examined. There is no one way to evaluate 
student learning, and librarians have struggled with this issue long before web-based instruction. 



Assessment can take many forms. Librarians could do a pretest and a posttest. It could be an 
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evaluation form distributed after the class or it could simply, be a question and answer session. 
These formats cannot be included in many web-based training programs due to constraints of the 
software. 

Usability studies are one way to make formal evaluations. Time, money and staffing are required. 

At the recent Association of College and Research Libraries Annual Conference in Detroit, 
Michigan in the presentation, "The User is the Expert, " three librarians reported on their 
experience using usability studies to inform gateway and tutorial web designs. The design should 
be user-centered and the methods to achieve can include the more costly but accurate testing 
methods of formal usability testing, focus groups, group testing and survey. Less costly and less 
accurate means are design walk-throughs, heuristic evaluation, card sorting, matching test or icon 
intuitiveness evaluation and field tests. Some of the results indicate that students tty to assign 
meaning to colors, navigation through frames is tricky, and links outside the tutorials are confusing. 

The IFLA Study 

The IFLA Study and presentation includes a detailed examination of these criteria while 
highlighting worldwide library websites that demonstrate effective use of these design features. The 
presentation will also review the library use of new web-based courseware products now on the 
market such as WebCT. 

The Web is not a static environment and so any guidelines drawn up for web-based instruction have 
to change with the technology. The fact that these guidelines are being updated is a healthy si^ 
that librarians are sharing the successes and the mistakes they have encountered when developing a 
web-based library. This is one of the best forms of evaluation. This is a sound progression and it is 
important that the focus of library instruction remains developing life-long learners and technology 
is the means to achieve this end. 
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Addendum: Standards and Evaluation Criteria Outside the United States 

The following are some projects outside the United States that are involved with teaching and 

learning in a networked environment: 

• The Higher Education community of the United Kingdom has an initiative called the The 
Electronic Libraries Programmes (eLib). (http://www.ukohi.ac.uk/services/elib/) They have 
posted standard guidelines and recommendations for the selection and use of standards in 
eLib projects, (http ://www.uko ln.ac.uk/services/elib/papers/ other/ standards/) 

• The Tavistock Institute, which undertakes evaluation studies, produced the document. 
Guidelines for eLib Project Evaluation. Prepared by John Kelleher, Elizabeth Sommerlad and 
Elliot Stem this document provides useful guidelines for developing and implementing 
evaluation in individual eLib projects. 

(http://www.ukoln.ac.Uk/services/elib/papers/tavistock/evaluation-guide/intro.html#defining .) 

• NetLinkS, a U.K. Training and Awareness project supported by the Joint Information 
Systems Committee of the Higher Education Funding Council of England. Initially funded 
for one year, fi'om September 1995, the project received further funding until May 1998. The 
NetLinkS Research Report is a detailed study of network learning. A link to other eLib 
projects throughout the United Kingdom is available fi'om their home page. 
(http://netwavs.shefac.uk/about/overview.htm) 

• Academic partnership in NLS resource design: a European case study by Jo Pye is a case 
study of European partner institutions concerned with electronic learning. This is a user needs 

■ analysis report addressing networked learner support in European partner institutions and 
development of an online course for librarians. This article was presented at the 2nd 
International Symposium on Networked Learner Support held in Sheffield, England in June 
1997. (http ://netwavs . shef ac . uk/rbase/papers/pve. htm) 

• DEDICATE is a good site with information on the development of distance education in 
Eastern Europe. The aim of the DEDICATE project is to develop cost-effective distance 
education courses in information literacy. The universities involved in this project are four 
library sites in Technological Universities in Estonia, Hungary, Latvia and Lithuania and at 
the International Center for Information Management, Systems, and Services, in Torun, 
Poland, (http ://educate . lib . chalmers. se/DEDIC ATE/ dedindex . html) 

• The Multimedia Education Unit at the University of Melbourne provides assistance to major 
educational projects aimed at transforming units, subjects or whole courses with the use of 
multimedia and educational technology. They are developing some distance education 
course, (http://www.meu.unimelb.edu.au/) 

• A web-based library education project worth looking at in Finland is one created by Irma 
Pasanen-Toumainen at Helsinki University of Technology. 
(http://www.hut.fi/Yksikot/Kiriasto/Palvelut/Koulutus/Informatiikka/) 
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